Spatially representing graphical interface elements as binaural audio content

ABSTRACT

Certain embodiments involve spatially representing graphical interface elements as binaural audio content. For example, a system accesses electronic content having various content elements and identifies the various content elements. The system generates a three-dimensional audio representation for a content element by: accessing an audio representation of the content element; identifying a visual location of the content element; and generating a three-dimensional audio representation of the content element by applying, to the audio representation of the content element and using the visual location, a finite impulse response filter using a head-related transfer function, wherein the head-related transfer function comprises a set of filter parameters that model the visual location of the content element and wherein, when played, the three-dimensional audio representation appears to originate from the visual location. The system can then output the three-dimensional audio representation.

TECHNICAL FIELD

This disclosure generally relates to accessibility tools and morespecifically relates to accessibility tools for spatially representinggraphical interface elements as binaural audio content.

BACKGROUND

Accessibility tools offer audio read-back of interfaces. In someinstances, accessibility tools can allow a user to access or useinformation and technology that may otherwise be inaccessible to theuser. For example, accessibility tools that offer audio read-backinterfaces can convert content or an element (e.g., text) displayed viaa display device to speech, which can allow a visually impaired user tointeract with the content or perceive the content displayed on thedisplay device. As an example, a visually impaired user can interactwith content via a mouse or keyboard (e.g., move a cursor to a positionof the content) and the accessibility tool can convert the content tospeech and provide an audio representation of the content (e.g., audiofeedback that reads the content to the user).

Current solutions for the audio being played back from such frameworksare monaural. For example, current monaural audio systems can provideaudio representations of an element via a single speaker or multiplespeakers. In this example, identical audio signals are provided to eachspeaker via the same audio channel. Therefore, existing monaural audiosystems provide audio representations that do not convey a perception orsensation of a location, depth, or position of the sound source, whichcan create the impression that all audio content comes from a particularpoint. Thus, two different audio representations of two differentelements would be perceived by a user as originating from the same pointin a space.

Existing accessibility tools therefore provide an experience that isone-dimensional from a user's perspective. As a result, an impaired usermay not experience a three-dimensional auditory sensation as compared toa user that is not impaired.

SUMMARY

Various embodiments of the present disclosure provide systems andmethods for spatially representing graphical interface elements asbinaural audio content.

In one example, a method for spatially representing graphical interfaceelements as binaural audio content includes: accessing, by a processor,electronic content having a plurality of content elements; identifying,by the processor, the plurality of content elements of the electroniccontent; and generating, by the processor, a three-dimensional audiorepresentation for a content element of the plurality of contentelements. Generating the three-dimensional audio representationincludes: accessing an audio representation of the content element;identifying a visual location of the content element; and generating athree-dimensional audio representation of the content element byapplying, to the audio representation of the content element and usingthe visual location, a finite impulse response filter using ahead-related transfer function, wherein the head-related transferfunction comprises a set of filter parameters that model the visuallocation of the content element and wherein, when played, thethree-dimensional audio representation appears to originate from thevisual location. The method further includes outputting, by theprocessor, the three-dimensional audio representation such that thethree-dimensional audio representation is accessible to a user devicethat displays the electronic content.

Additional features and advantages of exemplary embodiments of thepresent disclosure will be set forth in the description which follows,and in part will be obvious from the description, or will be learned bythe practice of such exemplary embodiments. The foregoing summary is notan extensive overview, and it is not intended to identify key elementsor indicate a scope. Rather the foregoing summary identifies aspects ofembodiments as a prelude to the detailed description presented below.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of an exemplary environment in which abinaural audio system generates binaural and spatial audiorepresentations of electronic content elements in accordance with one ormore embodiments.

FIG. 2 is a flow chart depicting an example of a process for generatingbinaural and spatial audio representations of electronic contentelements in accordance with one or more embodiments.

FIG. 3 is an example of a block diagram of a computing device thatexecutes a binaural audio system to generate binaural and spatial audiorepresentations of electronic content elements in accordance with one ormore embodiments.

DETAILED DESCRIPTION

Embodiments described herein include binaural audio applications andsystems that spatially represent graphical interface elements asbinaural audio content. A binaural audio application generates soundcues that represent the locations of elements in a graphical interface.For example, if an “edit” menu is located on the upper right-hand sideof a graphical interface, the binaural audio application generates acorresponding sound cue by playing an audio recording of the word “edit”for a user (e.g., when the user moves a cursor near the “edit” menu orotherwise interacts with the “edit” menu via user input). The sound cueis generated such that, from the perspective of the user, the audiosource sounds as if it is positioned at a location to the right of andabove the user's head (e.g., the user can perceive the sound asoriginating from a location corresponding to the location of the “edit”menu in the graphical interface). In this manner, an element of thegraphical interface is aurally represented in three dimensions. Thus,the binaural audio application provides improved accessibility byallowing a sight-impaired user to more easily distinguish between audiorepresentations of interface elements.

The following non-limiting example is provided to introduce certainembodiments. A binaural audio application accesses a web document to beaugmented with three-dimensional sound cues. The binaural audioapplication scans the web document and thereby identifies variousdocument elements (e.g., menus, buttons, text, etc.). The binaural audioapplication determines a location of each document element as displayedon the web document. The binaural audio also generates, for eachdocument element, a corresponding audio representation. For example, ifa document element includes text, the binaural audio applicationexecutes a text-to-speech engine to generate a corresponding audiorepresentation. In some examples, each document element is associatedwith a placeholder (e.g., a textual description of the element). In thisexample, the binaural audio application scans the web document andidentifies a placeholder associated with each identified documentelement. The placeholder can then be converted to an audiorepresentation of the document element by executing a text-to-speechalgorithm. To generate a binaural and spatial audio representation ofthe web document, the binaural audio application applies a finiteimpulse response (“FIR”) filter to each audio representation using ahead-related transfer function (“HRTF”). The FIR filter can include oneor more algorithms that can be applied to an audio signal orrepresentation to generate one or more three-dimensional audio signalsor sounds, which when played, cause the three-dimensional audio signalor sound to appear to originate from a source at a particular aurallocation with respect to the user (e.g., above and to the right, belowand to the left, behind and to the right, etc.). Each HRTF is configuredwith parameters that cause the aural location of a particular documentelement to correspond to the visual location of the particular documentelement (e.g., the location of the document element as displayed on theweb document).

Continuing with this example, the binaural audio application augmentsthe web document by adding three-dimensional audio accessibility contentto the web document. For example, the binaural audio applicationaugments the web document by associating each identified documentelement with a corresponding generated three-dimensional audio signalsor sounds, which allows the web document to be aurally modeled to a userif the web document is displayed in a graphical interface. For instance,a web browser or other end-user application that displays the webdocument can have an accessibility function used by visually impairedusers. If the augmented web document is accessed, the accessibilityfunction retrieves the audio accessibility content (e.g., generatedthree-dimensional audio signals associated with each identified documentelement of the web document). Each three-dimensional signal is played ifa particular document element is highlighted, selected, or otherwiseindicated by a user input by outputting corresponding binaural audiosignals.

These illustrative examples are given to introduce the reader to thegeneral subject matter discussed here and are not intended to limit thescope of the disclosed concepts. The following sections describe variousadditional features and examples with reference to the drawings in whichlike numerals indicate like elements, and directional descriptions areused to describe the illustrative examples but, like the illustrativeexamples, should not be used to limit the present disclosure.

As used herein, the term “audio representation” is used to refer to anyaudio value, audio signal, sound, or audio file.

FIG. 1 is a block diagram of an exemplary environment 100 in which abinaural audio system 102 generates binaural and spatial audiorepresentations of electronic content elements in accordance with one ormore embodiments. The environment 100 includes the binaural audio system102, one or more computing devices 104, and one or more data storageunits 112. The binaural audio system 102, the computing devices 104, andthe data storage unit 112 are communicatively coupled via one or moredata networks 108 (e.g., the Internet, one or more local area networks(“LAN”), one or more wired area networks, or some combination thereof).

In some embodiments, a user of the computing device 104 visits a webpageor an application store to explore applications supported by thebinaural audio system 102. The binaural audio system 102 provides theapplications as a software as service (“SaaS”), or as a standaloneapplication that may be installed on the computing device 104, or as acombination.

In some embodiments, the computing device 104 represents various typesof client devices. For example, the computing device 104 is a mobiledevice (e.g., a mobile telephone, a smartphone, a PDA, a tablet, alaptop, a camera, a tracker, a watch, a wearable device, etc.). Thecomputing device 104, however, in some embodiments, is a non-mobiledevice (e.g., a desktop computer or another type of client device). Inthis example, the binaural audio system 102 is implemented on, executedby, or stored on one or more computing devices 104. For example, thebinaural audio system 102 is stored on a memory device 106 of thecomputing device 104. In some embodiments, the binaural audio system 102is executed on the one or more computing devices 104 via a binauralaudio application 140.

In some embodiments, the binaural audio system 102 includes the binauralaudio application 140, which can include one or more instructions storedon a computer-readable storage medium and executable by one or moreprocessors of the computing device 104. When executed by the one or moreprocessors, the computer-executable instructions of the binaural audioapplication 140 cause the binaural audio system 102 to generate binauraland spatial audio representations of electronic content elements.

For example, a user of the computing device 104 accesses electroniccontent via the computing device 104. The electronic content caninclude, or be a part of, a graphical user interface displayed via adisplay device. Electronic content can include, but is not limited to, awebpage, website, or any electronic content (e.g., text, images, videos,animations, documents, user interfaces, etc.) In some embodiments, theelectronic content includes various electronic content elements (e.g.,menus, buttons, text, images, etc.).

In some embodiments, the binaural audio application 140 causes thebinaural audio system 102 to access the electronic content and scan theelectronic content to identify one or more electronic content elements.For example, the binaural audio system 102 accesses a webpage accessedby the user and scans the webpage to identify text, menus, buttons, orany graphical interface element of the webpage. In some embodiments,each electronic content element in the electronic content is associatedwith a placeholder such as, for example, a textual description orrepresentation of the electronic content element. In this example, thebinaural audio application 140 causes the binaural audio system 102 toscan the electronic content and identify one or more electronic contentelements and a placeholder associated with each identified electroniccontent element.

The binaural audio application 140 causes the binaural audio system 102to determine a position or location of an identified electronic contentelement. In some examples, the binaural audio system 102 determines aposition or location of an electronic content element as displayed inthe electronic content or as displayed in a graphical user interfacethat includes the electronic content. For example, the binaural audiosystem 102 identifies text on a webpage and identifies a visual locationor position of the text as displayed on the webpage or a graphical userinterface that includes the webpage. In some embodiments, the positionor location of an electronic content element is an azimuth and/orelevation value or position of the electronic content element. Anazimuth value or position can correspond to a position on a horizontalplane and an elevation value or position can correspond to a position ona vertical plane. In some embodiments, the binaural audio system 102determines a position or location of an electronic content element byaccessing the data storage unit 112, another database, or an electroniccontent object model that includes data indicating a location orposition of an electronic content element as displayed in the electroniccontent or as displayed in a graphical user interface that includes theelectronic content. As an example, the binaural audio system 102determines a position or location of text on a webpage by accessing adocument object model (“DOM”) that indicates a structure of the webpagesuch as, for example, positions of text and other content on the webpageand determining the position or location of the text on the webpage fromthe DOM. As another example, the data storage unit 112 includeselectronic content element position data 114 that indicates position orlocation data associated with various electronic content elements andthe binaural audio system 102 obtains or receives a position or locationof an identified electronic content element from the data storage unit112.

In some embodiments, the binaural audio application 140 causes thebinaural audio system 102 to access an audio representation of anidentified electronic content element. An audio representationassociated with an electronic content can include an audio signal or asound that, when played, indicates the electronic content element. Insome embodiments, the binaural audio application 140 causes the binauralaudio system 102 to access an audio representation of an electroniccontent element from the data storage unit 112 or any other source. Forexample, the data storage unit 112 stores audio representation data 116that indicates various audio representations associated with variouselectronic content elements and the binaural audio system 102 obtains orreceives an audio representation of an identified electronic contentelement from the data storage unit 112. As an example, if an electroniccontent element includes text, the binaural audio system 102 obtains orreceives an audio representation of the text from the data storage unit112. As another example, the electronic content is associated with aplaceholder such as, for example, a textual description orrepresentation of the electronic content element and the audiorepresentation data 116 indicates various audio representationsassociated with various placeholders. In this example, the binauralaudio system 102 obtains or receives an audio representation associatedwith a placeholder of an electronic content element from the datastorage unit 112. In some embodiments, the binaural audio application140 causes the binaural audio system 102 to generate an audiorepresentation of an identified electronic content element. As anexample, the electronic content is associated with a placeholder and thebinaural audio application 140 causes the binaural audio system 102 touse one or more text-to-speech algorithms to generate an audiorepresentation of the electronic content element based on theplaceholder. For example, the binaural audio system 102 uses the one ormore text-to-speech algorithms or engines to convert a textualdescription or representation of the electronic element to an audiorepresentation (e.g., audio file, audio value, etc.).

In some embodiments, the binaural audio application 140 causes thebinaural audio system 120 to generate a binaural and spatial audiorepresentation of an identified electronic content element based on theaudio representation of the electronic content element. In someexamples, a binaural and spatial audio representation of an electroniccontent element is a three-dimensional audio representation of theelectronic content element. The three-dimensional audio representationcan convey a perception or sensation of a location, depth, or positionof the electronic content element. For example, the three-dimensionalaudio representation of an electronic content element can convey aposition or location of the electronic content element as displayed inthe electronic content or as displayed in a graphical user interfacethat includes the electronic content such that, when played, thethree-dimensional audio representation is perceived by a user of thecomputing device 104 as originating from the position or location of theelectronic content element.

In some embodiments, the binaural audio application 140 causes thebinaural audio system 120 to generate a binaural and spatial audiorepresentation of an electronic content element by applying a finiteimpulse response (“FIR”) filter to an audio representation of theelectronic content element using a head-related transfer function(“HRTF”). The FIR filter can include one or more algorithms that can beapplied to an audio signal or representation to generate one or morebinaural and spatial audio representations, which when played, cause thebinaural and spatial audio representation to appear to originate from asource at a particular aural location with respect to the user (e.g.,above and to the right, below and to the left, behind and to the right,etc.). In some embodiments, the HRTF is configured with one or moreparameters or functions associated with a location or position (e.g.,azimuth or elevation positions). The parameter or function of the HRTFcan model a location or position and can be used to modify an audiosignal or representation to generate a binaural and spatial audiorepresentation that, when played, can be perceived as originating fromthe particular location or position. In some embodiments, the binauralaudio system 102 can generate a binaural audio representation of anelectronic content element by applying the FIR filter to an audiorepresentation of the electronic content element using a parameter orfunction associated with a location or position of the electroniccontent element. When played, the binaural audio representation of theelectronic content element can be perceived as originating from thevisual location of the electronic content element (e.g., the location orposition of the electronic content element as displayed in theelectronic content or as displayed in a graphical user interface thatincludes the electronic content).

In some embodiments, the data storage unit 112 includes HRTF data 118,which indicates various parameters or functions associated with variouslocations (e.g., azimuth or elevation positions). In this example, thebinaural audio system 102 can obtain or receive HRTF data 118 associatedwith a position or location of an identified electronic content elementand apply a FIR filter to the audio representation of the identifiedelectronic content using the HRTF data 118 to generate a binaural andspatial audio representation of the electronic content.

In some embodiments, the binaural audio application 140 causes thebinaural audio system 120 to augment the electronic content withgenerated binaural and spatial audio representations of variouselectronic content elements. For example, the binaural audio system 120augments the electronic content by associating each identifiedelectronic content element with a corresponding generated binaural andspatial audio representation. In this example, the binaural audioapplication 140 can cause the binaural audio system 120 to output one ormore of the generated binaural and spatial representations, which canallow the electronic content to be aurally modeled to a user if theelectronic content is displayed in a graphical interface via thecomputing device 104.

For instance, the user of the computing device 104 accesses theelectronic content via a web browser or other end-user application usingthe computing device 104. The computing device 104, the web browser, orend-user application can include an accessibility tool or function usedby visually impaired users. In this example, when the user accesses theelectronic content and/or provides user input accessing theaccessibility tool or function, the binaural audio system 120 retrievesaudio accessibility content (e.g., the generated binaural and spatialaudio representations associated with various electronic contentelements of the electronic content). Each binaural and spatial audiorepresentations is played if a particular electronic content element ishighlighted, selected, or otherwise indicated by a user input byoutputting the corresponding binaural and spatial audio representation.

In some embodiments, the computing device 104 can include one or moresensors that can detect a direction of the user's field of view withrespect to electronic content or electronic content elements displayedvia the computing device 104. The sensor can include a camera or beincorporated into a camera. The sensor can be configured to capture animage of the eye of the user and the binaural audio application 140 cancause the binaural audio system 102 to determine the direction of thefield of view of the user relative to the electronic content based atleast in part on the image by using various image processing methods andtechniques. In another embodiment, the one or more sensors can beconfigured to monitor movements of an eye of the user or muscles near aneye of the user of the computing device 104 and the binaural audiosystem 102 can determine the direction of the user's field of viewrelative to electronic content based at least in part on the monitoredmovements. In still another embodiment, the sensor may be configured tomonitor or measure electrical activity of muscles moving the eye of theuser of the computing device 104 and the binaural audio system 102 canbe configured to determine the direction of the user's field of viewrelative to the electronic content displayed via the computing device104. In some embodiments, the sensor may detect the user's eye gaze,line-of sight, or field of view through various methods and techniques,including, for example, analyzing the user's body or head posture. As anexample, the sensor can include a head-mounted display or a head-mountedsensor for detecting a motion of the user's head or for detecting theuser's head posture and transmitting data about the motion of the user'shead or data about the user's head posture to the binaural audio system102, which can determine the direction of the field of view of the userof the computing device 104 based on the data.

In some examples, the binaural audio application 140 can cause thebinaural audio system 120 to output one or more generated binaural andspatial representations in response to determining that the user of thecomputing device 104 is looking at, or in the direction of, a particularelectronic content element. For example, if the user is looking at texton a webpage, the binaural audio system 102 can output a correspondingbinaural and spatial audio representation. In this manner, the binauralaudio system 102 can generate and output binaural and spatial audiobased on a direction of a field of view of a user of the computingdevice 104.

In the example depicted in FIG. 1, a user can interface with the one ormore user devices 110 to access the binaural audio system 102. In someembodiments, each of the user devices 110 represents various types ofclient devices. For example, the user device 110 is a mobile device(e.g., a mobile telephone, a smartphone, a PDA, a tablet, a laptop, acamera, a tracker, a watch, a wearable device, etc.). The user device110, however, in some embodiments, is a non-mobile device (e.g., adesktop computer or another type of client device). In some embodiments,the binaural audio system 102 is executed on the one or more userdevices 110 via a binaural audio application 140. In this example, theuser device 110 includes one or more components of the binaural audiosystem 102.

Although the exemplary environment 100 of FIG. 1 is depicted as having acertain number of components, in other embodiments, the exemplaryenvironment 100 has any number of additional or alternative components.Further, while FIG. 1 illustrates a particular arrangement of thecomputing device 104, the binaural audio system 102, user devices 110,and the data storage unit 112, various additional arrangements arepossible. As an example, while FIG. 1 illustrates data storage unit 112and the binaural audio system 102 as part of separate systems, in someembodiments, the data storage unit 112 and the binaural audio system 102are part of a single system.

FIG. 2 is a flow chart depicting an example of a process for generatingbinaural and spatial audio representations of electronic contentelements in accordance with one or more embodiments. In someembodiments, one or more processing devices such as, for example, thecomputing system described herein with respect to FIG. 3, implementoperations depicted in FIG. 2 by executing suitable program code (e.g.,the binaural audio system 102 of FIG. 1) that implements one or morealgorithms encompassed by the process 200. For illustrative purposes,the process 200 is described with reference to the examples depicted inFIG. 1, but other implementations are possible.

In block 202, electronic content that includes various electroniccontent elements is accessed. For example, a user of a computing device104 accesses electronic content via the computing device 104. Theelectronic content can include, or be a part of, a graphical userinterface displayed via a display device. Electronic content caninclude, but is not limited to, a webpage, website, or any electroniccontent (e.g., text, images, videos, animations, documents, userinterfaces, etc.) In some embodiments, the electronic content includesvarious electronic content elements (e.g., menus, buttons, text, images,icons, etc.).

In some embodiments, a binaural audio system 102 is implemented on,executed by, or stored on the computing device 104. In some embodiments,the binaural audio system 102 includes a binaural audio application 140,which can include one or more instructions stored on a computer-readablestorage medium and executable by processors of the computing device 104.When executed by the one or more processors, the computer-executableinstructions of the binaural audio application 140 can cause thebinaural audio system 102 to access the electronic content accessed bythe user.

In block 204, the various electronic content elements of the electroniccontent are identified. In some embodiments, the binaural audioapplication 140 causes the binaural audio system 102 to access theelectronic content and scan the electronic content to identify one ormore electronic content elements. For example, if the electronic contentis a webpage accessed by a user of the computing device 104, thebinaural audio system 102 accesses the webpage and scans the webpage toidentify text, menus, buttons, or any graphical interface elements ofthe webpage. In some embodiments, the binaural audio system 102 canidentify one or more electronic content elements by accessing adatabase, a model, or any other source that includes data indicating theelectronic content elements included in the electronic content elementand identifying the electronic content elements.

In some embodiments, an electronic content element in the electroniccontent is associated with a placeholder such as, for example, a textualdescription or representation of the electronic content element. In someexamples, the binaural audio system 102 can obtain or receive dataindicating a placeholder associated with an electronic content elementfrom the electronic content (e.g., if the data is embedded within theelectronic content), the data storage unit 112, another computingdevice, user input (e.g., if a user programs the binaural audio system102 to include the data), or any other source. In this example, thebinaural audio application 140 causes the binaural audio system 102 toscan the electronic content, identify one or more electronic contentelements, and obtain data indicating a placeholder associated with eachidentified electronic content element.

In block 206, an audio representation of an identified electroniccontent element is accessed. In some embodiments, the binaural audioapplication 140 causes the binaural audio system 102 to access the audiorepresentation of the electronic content element. The audiorepresentation of the electronic content element can include an audiosignal, audio value, or sound that indicates the identified electroniccontent element. As an example, if the electronic content elementincludes text, the audio representation can include an audio signal orvalue that, when played, corresponds to the text.

In some embodiments, the binaural audio application 140 causes thebinaural audio system 102 to access the audio representation from thedata storage unit 112 or any other source. As an example, the datastorage unit 112 includes audio representation data 116 that indicatesvarious audio representations associated with various electronic contentelements. The binaural audio system 102 accesses the audiorepresentation data 116 and obtains the audio representation associatedwith the identified electronic content element. In another example, thebinaural audio system 102 receives data indicating an audiorepresentation associated with the identified content element from thedata storage unit 112. In some embodiments, the electronic content isassociated with a placeholder (e.g., a textual representation ordescription of the electronic content element). In this example, thebinaural audio application 140 causes the binaural audio system 102 toaccess an audio representation associated with the placeholder of theelectronic content. For example, the audio representation data 116indicates various audio representations associated with variousplaceholders of electronic content elements and the binaural audiosystem 102 obtains or receives the audio representation associated withthe placeholder of the identified electronic content element from thedata storage unit 112.

In some embodiments, the binaural audio application 140 causes thebinaural audio system 102 to generate an audio representation of anidentified electronic content element. For example, the identifiedelectronic content is associated with a placeholder such as, forexample, a textual representation of the electronic content element, andthe binaural audio application 140 causes the binaural audio system 102to use one or more text-to-speech algorithms to generate an audiorepresentation of the electronic content element. For example, thebinaural audio system 102 uses the one or more text-to-speech algorithmsto convert the textual description or representation of the electronicelement to an audio representation (e.g., audio file, audio value,etc.).

In block 208, a location of the identified electronic content element isidentified. In some embodiments, the binaural audio application 140causes the binaural audio system 102 to identify the location orposition of the electronic content element. The location of theelectronic content element can correspond to a position or location ofthe electronic content element as displayed in the electronic content oras displayed in a graphical user interface. For example, the location ofthe electronic content element can correspond to a visual location ofthe electronic content element such as, for example, a visual locationof a menu or text within a webpage or within a graphical user interfacedisplaying the webpage. In some embodiments, the binaural audio system102 determines the position or location of the electronic contentelement by determining an azimuth (e.g., horizontal) and/or elevation(e.g., vertical) value or position of the electronic content element.

In some embodiments, the binaural audio system 102 determines a positionor location of the electronic content element by accessing a database oran electronic content object model that includes data indicatinglocations or positions of one or more electronic content elements asdisplayed in the electronic content or as displayed in a graphical userinterface that includes the electronic content. As an example, thebinaural audio system 102 determines a position or location of text on awebpage by accessing a document object model (“DOM”) that indicates astructure of the webpage such as, for example, positions of text andother content on the webpage and determining the position or location ofthe text on the webpage. As another example, the data storage unit 112includes electronic content element position data 114 that indicatesposition or location data associated with various electronic contentelements and the binaural audio system 102 obtains or receives theposition or location of the electronic content element from the datastorage unit 112.

In some embodiments, the binaural audio system 102 determines theposition or location of the electronic content element based on agraphical user interface that includes the electronic content element ora display device via which electronic content is output. For example,the binaural audio system 102 determines a size or dimension of thegraphical user interface and determines a position of the electroniccontent element relative to the size or dimension of the graphical userinterface. As an example, the binaural audio system 102 determinesdimensions of the graphical user interface and determines a position ofthe electronic content element relative to a top, bottom, left, or rightborder or boundary of the graphical user interface. In this example, thebinaural audio system 102 uses the relative position of the electroniccontent element to determine a relative azimuth and/or relativeelevation value or position of the electronic content element withregard to the graphical user interface. In some examples, the binauralaudio system 102 determines a position or location of the electroniccontent element based on a display device of the computing device 104via which the electronic content is output in substantially the samemanner as described above. For example, the binaural audio system 102obtains data indicating a size or dimensions of the display device anddetermines a position of the electronic content element relative to atop, bottom, left, or right border or boundary of the display device anduses the relative position of the electronic content element todetermine a relative azimuth and/or relative elevation value or positionof the electronic content element with regard to the display device.

In block 210, a three-dimensional audio representation of the electroniccontent element is generated. In some embodiments, the binaural audioapplication 140 causes the binaural audio system 102 to generate thethree-dimensional audio representation of the electronic content elementusing the audio representation of the electronic content element (e.g.,the audio representation accessed in block 206) and the location of theelectronic content element (e.g., the location determined in block 208).The three-dimensional audio representation of the electronic contentelement can be a binaural and spatial audio representation that canconvey a perception or sensation of a location, depth, or position ofthe electronic content element (e.g., the location or position of theelectronic content element identified in block 208). For example, if theelectronic content includes text located toward the left on a webpage,the three-dimensional the binaural audio system 102 generates an audiorepresentation of the text that, when played, is perceived by a user asoriginating from the left of the webpage.

In some embodiments, the binaural audio application 140 causes thebinaural audio system 120 to generate the three-dimensional audiorepresentation of the electronic content element using various methodsor techniques. For example, the binaural audio system 102 generates thethree-dimensional audio representation by applying a FIR filter to theaudio representation of the electronic content element (e.g., the audiorepresentation accessed or generated in block 206). The binaural audiosystem 102 can use a HRTF to apply the FIR filter to the audiorepresentation of the electronic content element to generate thethree-dimensional audio representation. The HRTF and/or FIR filter canuse one or more algorithms or functions that can be applied to an audiosignal, audio value, or audio representation to generate or synthesize athree-dimensional audio representation that, when played, can beperceived by a user as originating or coming from a particular point inspace.

In some embodiments, the HRTF includes one or more parameters orfunctions that can be used to generate or synthesize thethree-dimensional audio representation. The one or more parameters orfunctions can be based on a location or position such that theparameters or functions can be used to generate a three-dimensionalaudio representation that when played, the three-dimensional audiorepresentation can be perceived as originating from the visual locationof the electronic content element (e.g., the location or position of theelectronic content element as displayed in the electronic content or asdisplayed in a graphical user interface that includes the electroniccontent).

In some embodiments, the binaural audio system 102 obtains or receivesHRTF data 118 from the data storage unit 112 for generating thethree-dimensional audio representation of the electronic contentelement. The HRTF data 118 can include data that indicates variousparameters or functions associated with various locations or positions(e.g., azimuth or elevation positions). In this example, the binauralaudio system 102 can obtain or receive HRTF data 118 associated with theposition or location of the electronic content element and apply a FIRfilter to the audio representation of the electronic content to generatethe three-dimensional audio representation of the electronic content. Inanother example, the binaural audio system 102 can obtain or receiveHRTF data 118 from any other source.

In some embodiments, the binaural audio system 102 determines theposition or location of the electronic content element relative to theposition or location of another electronic content element and generatesa three-dimensional audio representation of each of the electronicelements based on the relative position or location of the electroniccontent element.

For example, the electronic content includes a first electronic contentelement, a second electronic content element, and a third electroniccontent element. The binaural audio system 102 determines a position orlocation of each of the first, second, and third electronic contentelements and compares the position or location of the electronic contentelements to determine a relative location or position of each electroniccontent element. As an example, the binaural audio system 102 compares alocation of the first electronic content element and a location of thesecond electronic content element and determines a distance between thefirst and second electronic content elements. The binaural audio system102 also compares a location of the first electronic element and alocation of the third electronic element and determines a distancebetween the first and third electronic elements. In this example, thebinaural audio system 102 compares the distance between the first andsecond electronic elements and the distance between the first and thirdelectronic elements and determines that the distance between the firstand second electronic elements is greater than the distance between thefirst and third electronic elements (e.g., the first electronic elementis closer to the third electronic element than the second electronicelement). The binaural audio system 102 can generate a three-dimensionalaudio representation of each of the first, second, and third electroniccontent elements based on the relative position of each electroniccontent element. For example, when the three-dimensional audiorepresentations of each of the first, second, and third electroniccontent elements are played, the three dimensional-audio representationof the first electronic content element is perceived as originating froma location that is close to a location from which the three-dimensionalaudio representation of the third electronic content element originates.

In block 212, the three-dimensional audio representation is output. Insome embodiments, the binaural audio application 140 causes the binauralaudio system 120 to output the three-dimensional audio representation.

For example, the binaural audio application 140 causes the binauralaudio system 120 to augment the electronic content with the generatedthree-dimensional audio representation of the electronic content element(e.g., the three-dimensional audio representation generated in block210). In some embodiments, the binaural audio application 140 augmentsthe electronic content element by associating an identified electroniccontent element (e.g., an electronic content element identified in block204) with a corresponding generated three-dimensional audiorepresentation, which can allow the electronic content to be aurallymodeled for a user when the electronic content is displayed in agraphical interface via the computing device 104.

For instance, the user of the computing device 104 accesses theelectronic content via a web browser or other end-user application usingthe computing device 104. The computing device 104, the web browser, orend-user application can include an accessibility tool or function usedby visually impaired users. In this example, when the user accesses theelectronic content and/or provides user input accessing theaccessibility tool or function, the binaural audio system 120 retrievesa generated three-dimensional audio representation associated with anelectronic content element (e.g., the three-dimensional audiorepresentation generated in block 210) and outputs the three-dimensionalaudio representation if the electronic content element is highlighted,selected, or if the user provides any user input to interact with theelectronic content element (e.g., moves a cursor near the electroniccontent element, provides touch input near the electronic contentelement etc.). Outputting the three-dimensional audio representation ofthe electronic content element can convey a position or location of theelectronic content element as displayed in the electronic content or asdisplayed in a graphical user interface that includes the electroniccontent. For example, when played, the three-dimensional audiorepresentation of the electronic content element is perceived by a userof the computing device 104 as originating from the position or locationof the electronic content element.

In some embodiments, in block 212, the binaural audio application 140causes the binaural audio system 120 to output the three-dimensionalaudio representation based on a direction of a field of view of the userof the computing device 104. For example, the binaural audio application140 can cause the binaural audio system 120 to determine that the userof the computing device 104 is looking at, or in the direction of, theelectronic content element (e.g., based on sensor data from a sensorconfigured to a field of view or a direction of an eye gaze of theuser). In this example, the binaural audio application 140 causes thebinaural audio system 120 to output the corresponding three-dimensionalaudio representation of the electronic content element in response todetermining that the user is looking at, or toward, the electroniccontent element.

System Implementation Example

Any suitable computing system or group of computing systems can be usedfor performing the operations described herein. FIG. 3 is an example ofa block diagram of a computing device that executes a binaural audiosystem 102 to perform the operations described herein.

The depicted example of the computing device 104 includes one or moreprocessors 302 communicatively coupled to one or more memory devices304. The processor 302 executes computer-executable program code storedin the memory device 304, accesses information stored in the memorydevice 304, or both. Examples of the processor 302 include amicroprocessor, an application-specific integrated circuit (“ASIC”), afield-programmable gate array (“FPGA”), or any other suitable processingdevice. The processor 302 can include any number of processing devices,including one or more processors 302 that are configured by program codeto implement the operations described above, such as the operationsdepicted in FIG. 2 that are described with respect to processingdevices.

The memory device 304 includes any suitable non-transitorycomputer-readable medium for storing the binaural audio system 102. Thecomputer-readable medium can include any electronic, optical, magnetic,or other storage device capable of providing a processor withcomputer-readable instructions or other program code. Non-limitingexamples of a computer-readable medium include a magnetic disk, a memorychip, a ROM, a RAM, an ASIC, optical storage, magnetic tape or othermagnetic storage, or any other medium from which a processing device canread instructions. The instructions may include processor-specificinstructions generated by a compiler or an interpreter from code writtenin any suitable computer-programming language, including, for example,C, C++, C#, Visual Basic, Java, Python, Perl, JavaScript, andActionScript. One or more memory devices 304 are used to implement theoperations described above, such as the operations depicted in FIGS. 1-2that are described with respect to one or more non-transitorycomputer-readable media.

The computing device 104 may also include a number of external orinternal devices such as input or output devices. For example, thecomputing device 104 is shown with an input/output (“I/O”) interface 308that can receive input from input devices or provide output to outputdevices. A bus 306 can also be included in the computing device 104. Thebus 306 can communicatively couple one or more components of thecomputing device 104. In some embodiments, the bus 306 is used toimplement the operations described above with respect to FIGS. 1-2 thatinvolve communicating signals via a data bus.

The computing device 104 executes program code that configures theprocessor 302 to perform one or more of the operations described abovewith respect to FIGS. 1-2. The program code includes, for example,binaural audio application 140 or other suitable applications thatperform one or more operations described herein. The program code may beresident in the memory device 304 or any suitable computer-readablemedium and may be executed by the processor 302 or any other suitableprocessor. In some embodiments, the program code described above isstored in the memory device 304, as depicted in FIG. 3. In additional oralternative embodiments, the program code described above is stored inone or more memory devices accessible via a data network.

The computing device 104 accesses the electronic content elementposition data 114, the audio representation data 116, or the HRTF data118 in any suitable manner. In some embodiments, the electronic contentelement position data 114, the audio representation data 116, or theHRTF data 118 is stored in one or more memory devices accessible via adata network 108. In additional or alternative embodiments, some or allof the electronic content element position data 114, the audiorepresentation data 116, or the HRTF data 118 is stored in the memorydevice 304.

The computing device 104 depicted in FIG. 3 also includes at least onenetwork interface 310. The network interface 310 includes any device orgroup of devices suitable for establishing a wired or wireless dataconnection to one or more data networks 108. Non-limiting examples ofthe network interface 310 include an Ethernet network adapter, a modem,and/or the like. The computing device 104 is able to communicate withone or more web servers 312 via which a user may access the binauralaudio system 102 or binaural audio application 140. In some embodiments,the network interface 310 is used to implement the operations describedabove with respect to FIGS. 1-2 that involve communicating signals via adata network.

General Considerations

Numerous specific details are set forth herein to provide a thoroughunderstanding of the claimed subject matter. However, those skilled inthe art will understand that the claimed subject matter may be practicedwithout these specific details. In other instances, methods,apparatuses, or systems that would be known by one of ordinary skillhave not been described in detail so as not to obscure claimed subjectmatter.

Unless specifically stated otherwise, it is appreciated that throughoutthis specification discussions utilizing terms such as “processing,”“computing,” “calculating,” “determining,” and “identifying” or the likerefer to actions or processes of a computing device, such as one or morecomputers or a similar electronic computing device or devices, thatmanipulate or transform data represented as physical electronic ormagnetic quantities within memories, registers, or other informationstorage devices, transmission devices, or display devices of thecomputing platform.

The system or systems discussed herein are not limited to any particularhardware architecture or configuration. A computing device can includeany suitable arrangement of components that provide a result conditionedon one or more inputs. Suitable computing devices include multipurposemicroprocessor-based computer systems accessing stored software thatprograms or configures the computing system from a general purposecomputing apparatus to a specialized computing apparatus implementingone or more embodiments of the present subject matter. Any suitableprogramming, scripting, or other type of language or combinations oflanguages may be used to implement the teachings contained herein insoftware to be used in programming or configuring a computing device.

Embodiments of the methods disclosed herein may be performed in theoperation of such computing devices. The order of the blocks presentedin the examples above can be varied—for example, blocks can bere-ordered, combined, and/or broken into sub-blocks. Certain blocks orprocesses can be performed in parallel.

The use of “adapted to” or “configured to” herein is meant as open andinclusive language that does not foreclose devices adapted to orconfigured to perform additional tasks or steps. Additionally, the useof “based on” is meant to be open and inclusive, in that a process,step, calculation, or other action “based on” one or more recitedconditions or values may, in practice, be based on additional conditionsor values beyond those recited. Headings, lists, and numbering includedherein are for ease of explanation only and are not meant to belimiting.

While the present subject matter has been described in detail withrespect to specific embodiments thereof, it will be appreciated thatthose skilled in the art, upon attaining an understanding of theforegoing, may readily produce alterations to, variations of, andequivalents to such embodiments. Accordingly, it should be understoodthat the present disclosure has been presented for purposes of examplerather than limitation, and does not preclude the inclusion of suchmodifications, variations, and/or additions to the present subjectmatter as would be readily apparent to one of ordinary skill in the art.

1. A method for spatially representing graphical interface elements asbinaural audio content, the method comprising: accessing, by aprocessor, electronic content having a plurality of content elements;identifying, by the processor, the plurality of content elements of theelectronic content; generating, by the processor, a three-dimensionalaudio representation for a content element of the plurality of contentelements, wherein generating the three-dimensional audio representationcomprises: accessing an audio representation of the content element;identifying a visual location of the content element by; accessing, bythe processor, a content object model specifying visual locations of theplurality of content elements when displayed in a graphical interface;and identifying, by the processor and from the content object model, thevisual location of the content element; and generating athree-dimensional audio representation of the content element byapplying, to the audio representation of the content element and usingthe visual location, a finite impulse response filter using ahead-related transfer function, wherein the head-related transferfunction comprises a set of filter parameters that model the visuallocation of the content element and wherein, when played, thethree-dimensional audio representation appears to originate from thevisual location; and outputting, by the processor, the three-dimensionalaudio representation such that the three-dimensional audiorepresentation is accessible to a user device that displays theelectronic content.
 2. The method of claim 1, wherein accessing theaudio representation of the content element comprises: obtaining, by theprocessor, a textual representation of the content element; andconverting, by the processor, the textual representation to the audiorepresentation.
 3. The method of claim 2, wherein converting the textualrepresentation to the audio representation comprises converting, by theprocessor, the textual representation to the audio representation byusing a text-to-speech engine to generate the audio representation fromthe textual representation.
 4. (canceled)
 5. The method of claim 1,wherein outputting the three-dimensional audio representation comprises:receiving, by the processor, user input to interact with the contentelement; and outputting, by the processor, the three-dimensional audiorepresentation in response to receiving the user input.
 6. The method ofclaim 1, wherein identifying the visual location of the content elementcomprises: determining, by the processor, an azimuth value of thecontent element; and determining, by the processor, an elevation valueof the content element.
 7. The method of claim 6, wherein generating thethree-dimensional audio representation of the content element comprise:applying, to the audio representation of the content element and usingthe azimuth value and the elevation value, the finite impulse responsefilter using the head-related transfer function, wherein the set offilter parameters are based on the azimuth value and the elevationvalue.
 8. A system comprising: a processing device; and a non-transitorycomputer-readable medium communicatively coupled to the processingdevice, wherein the processing device is configured to performoperations comprising: accessing electronic content having a pluralityof content elements; identifying the plurality of content elements ofthe electronic content; generating a three-dimensional audiorepresentation for a content element of the plurality of contentelements, wherein generating the three-dimensional audio representationcomprises: accessing an audio representation of the content element;identifying a visual location of the content element by: accessing acontent object model specifying visual locations of the plurality ofcontent elements when displayed in a graphical interface; andidentifying the visual location of the content element from the contentobject model; and generating a three-dimensional audio representation ofthe content element by applying, to the audio representation of thecontent element and using the visual location, a finite impulse responsefilter using a head-related transfer function, wherein the head-relatedtransfer function comprises a set of filter parameters that model thevisual location of the content element and wherein, when played, thethree-dimensional audio representation appears to originate from thevisual location; and outputting the three-dimensional audiorepresentation such that the three-dimensional audio representation isaccessible to a user device that displays the electronic content.
 9. Thesystem of claim 8, wherein the processing device is further configuredto access the audio representation of the content element by: obtaininga textual representation of the content element; and converting thetextual representation to the audio representation.
 10. The system ofclaim 9, wherein the processing device is further configured to convertthe textual representation to the audio representation by converting thetextual representation to the audio representation by using atext-to-speech engine to generate the audio representation from thetextual representation.
 11. (canceled)
 12. The system of claim 8,wherein the processing device is further configured to output thethree-dimensional audio representation by: receiving user input tointeract with the content element; and outputting the three-dimensionalaudio representation in response to receiving the user input.
 13. Thesystem of claim 8, wherein the processing device is further configuredto identify the visual location of the content element by: determiningan azimuth value of the content element; and determining an elevationvalue of the content element.
 14. The system of claim 13, wherein theprocessing device is further configured to generate thethree-dimensional audio representation of the content element byapplying, to the audio representation of the content element and usingthe azimuth value and the elevation value, the finite impulse responsefilter using the head-related transfer function, wherein the set offilter parameters are based on the azimuth value and the elevationvalue.
 15. A system comprising: a means for accessing electronic contenthaving a plurality of content elements; a means for identifying theplurality of content elements of the electronic content; a means forgenerating a three-dimensional audio representation for a contentelement of the plurality of content elements, wherein generating thethree-dimensional audio representation comprises: accessing an audiorepresentation of the content element; identifying a visual location ofthe content element by: accessing a content object model specifyingvisual locations of the plurality of content elements when displayed ina graphical interface; and identifying the visual location of thecontent element from the content object model; and generating athree-dimensional audio representation of the content element byapplying, to the audio representation of the content element and usingthe visual location, a finite impulse response filter using ahead-related transfer function, wherein the head-related transferfunction comprises a set of filter parameters that model the visuallocation of the content element and wherein, when played, thethree-dimensional audio representation appears to originate from thevisual location; and a means for outputting the three-dimensional audiorepresentation such that the three-dimensional audio representation isaccessible to a user device that displays the electronic content. 16.The system of claim 15, further comprising: a means for accessing theaudio representation of the content element by: obtaining a textualrepresentation of the content element; and converting the textualrepresentation to the audio representation.
 17. The system of claim 16,further comprising a means for converting the textual representation tothe audio representation by: converting the textual representation tothe audio representation using a text-to-speech engine to generate theaudio representation from the textual representation.
 18. (canceled) 19.The system of claim 16, further comprising a means for outputting thethree-dimensional audio representation by: receiving user input tointeract with the content element; and outputting the three-dimensionalaudio representation in response to receiving the user input.
 20. Thesystem of claim 16, further comprising a means for generating thethree-dimensional audio representation of the content element by:determining an azimuth value of the content element; determining anelevation value of the content element; and applying, to the audiorepresentation of the content element and using the azimuth value andthe elevation value, the finite impulse response filter using thehead-related transfer function, wherein the set of filter parameters arebased on the azimuth value and the elevation value.